Skip to content

[RTOP-76] runtime logs parser#13

Merged
lucasbutzke merged 56 commits into
developmentfrom
RTOP-76-Runtime-logs-parser
Oct 8, 2025
Merged

[RTOP-76] runtime logs parser#13
lucasbutzke merged 56 commits into
developmentfrom
RTOP-76-Runtime-logs-parser

Conversation

@lucasbutzke
Copy link
Copy Markdown
Contributor

Summary

This PR refactors the logging system to provide a consistent JSON-based format and simplifies the API for accessing buffered logs. The logging internals have been streamlined so consumers interact with a cleaner and more predictable interface.

Changes

  1. Logging Refactor
  • Unified parsing and formatting logic into LogParser.
  • Replaced raw Python logging-formatted strings with structured JSON logs.
  • Integrated timestamp normalization (UNIX → ISO8601 UTC).
  • Standardized log schema:
{
  "timestamp": "2025-10-03T19:02:58Z",
  "level": "INFO",
  "message": "Scan Count: 0"
}
  1. API Updates
  • BufferedHandler.get_logs() now returns structured JSON objects instead of raw log strings.
  • Added utility normalize_buffer_logs() for transforming existing raw logs into JSON.
  • Simplified get_logger() usage by ensuring consistent return types (logger, buffer).

Why
Previously, logs in the buffer contained mixed Python metadata and embedded JSON, which made them difficult to parse programmatically. This refactor ensures logs are both human-readable and machine-friendly, and the updated API makes log consumption straightforward for downstream services.
Impact
Developers can now rely on a stable JSON schema for log processing.
API consumers no longer need to manually parse log lines.
Backward compatibility: minor change — consumers expecting raw log strings must now handle structured JSON.

marconetsf and others added 30 commits September 11, 2025 14:42
Introduces a plugin driver framework supporting both native and Python plugins, including configuration parsing, driver management, and integration into the main PLC application. Updates CMake to link Python libraries and adds example configuration files for plugins. (WIP)
Removed legacy driver_api files and introduced new plugin driver structures to support Python-based plugins. Added a simple_modbus.py driver and configuration for Modbus slave integration. Updated plugin_driver.h and python_plugin_bridge.h to support Python plugin bindings and runtime argument passing. Modified plc_main.c to call plugin_driver_init instead of plugin_driver_start. Updated plugins.conf to use the new Python Modbus plugin.
Accordingly with the documentation, python header should be the first called and for security, we have to define PY_SSIZE_T_CLEAN
Everything accordingly BARR Standard
everything accordingly BARR standard
deleting python cycle function since it will be running async
for some reason, when init is called it can successfully link buffers and function address between runtime and plugin, but when the same previous parsed "struct" is called within start function, the buffer pointer was no longer pointing to the right address.
Corrects how bool_output buffer values are read and written by accessing the actual value via .contents.value instead of the pointer. Updates example plugin to use SafeBufferAccess for safer buffer operations and improves output logging.
Stop is already being called within destroy function
Eliminated the _runtime_args_capsule global variable and related code from example_python_plugin.py, simplifying state management and usage of runtime arguments.
Moved plugin thread creation to Python side and removed native thread management for Python plugins. Updated function names in python_binds_t for clarity. Improved plugin start, stop, and cleanup logic to use Python-side functions and ensured proper GIL handling. Cleaned up resource management and removed unused thread fields from plugin_instance_t.
Improves Python GIL state management in plugin_driver by using a static variable and ensuring proper acquisition/release during plugin lifecycle. Moves plugin driver cleanup earlier in plc_main to avoid double destruction and adds more informative logging during plugin stop.
The plugin's main loop now runs in a background thread using Python's threading module. Added a stop event to allow graceful termination of the loop in stop_loop, improving plugin lifecycle management and preventing blocking the main thread.
Replaces manual ctypes structure and buffer access with type-safe wrappers from python_plugin_types. Updates OpenPLCModbusDataBlock to use SafeBufferAccess for reading and writing coil values, improving safety and error handling. Refactors plugin initialization and server startup for better diagnostics and reliability.
Revised README to clarify plugin type values, enhance Python plugin type safety, and document usage of the new python_plugin_types.py module. Added examples for thread-safe buffer access, advanced Modbus implementation, and improved plugin initialization with structure validation and error handling.
Made plugin_mutex_take and plugin_mutex_give functions public in plugin_driver.h and used them to protect buffer access in the PLC cycle thread. Also refactored plugin_driver variable to be global in plc_main.c for thread safety.
…er-contracts

Task/rtop 57 implement driver contracts
* thread safe and dump access in the same function

* adding batch functions that allows multiple reads/writes

* Providing wrappers for IS, IR and HR

* avoiding magical numbers

* avoiding generic exception handling
@lucasbutzke lucasbutzke requested review from Copilot and thiagoralves and removed request for Copilot and thiagoralves October 3, 2025 19:20
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements a comprehensive refactor of the logging system to provide a unified JSON-based format with a cleaner API, transitioning from mixed Python logging with embedded JSON to structured JSON logs. Additionally, it introduces a comprehensive plugin system architecture for external drivers and communication protocols.

Key Changes:

  • Logging System Refactor: Replaced raw Python logging strings with structured JSON output, unified parsing/formatting logic into LogParser, and simplified the BufferedHandler API
  • Plugin System Architecture: Added complete plugin driver system supporting both Python and native C plugins with thread-safe buffer access and virtual environment support
  • Development Infrastructure: Enhanced build system, virtual environment management, and containerization with proper script execution

Reviewed Changes

Copilot reviewed 39 out of 41 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
webserver/logger/* New modular logging system with JSON formatting, buffered handlers, and unified parsing
core/src/drivers/* Complete plugin system implementation with Python/C plugin support and thread-safe buffer access
webserver/*.py Refactored to use new logger system, commented out old logging calls
scripts/* Added plugin virtual environment management and enhanced installation scripts
install.sh/start_openplc.sh Improved installation and startup processes with proper dependency management
Comments suppressed due to low confidence (2)

webserver/logger/parser.py:1

  • [nitpick] There's inconsistent indentation in the commented code. The comment on line 77 should either be properly aligned or the entire commented block should be cleaned up for better readability.
# logger/parser.py

webserver/logger/bufferhandler.py:1

  • [nitpick] There are malformed comments with inconsistent indentation. Clean up the commented code or remove it entirely to improve code readability.
import logging

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment thread webserver/logger/logger.py
Comment thread webserver/plcapp_management.py
Comment thread core/src/drivers/plugins/python/examples/example_python_plugin.py Outdated
Comment thread webserver/runtimemanager.py
Comment thread core/src/drivers/plugin_driver.c
@lucasbutzke lucasbutzke changed the base branch from development to RTOP-52-Standard-Network-Driver October 7, 2025 15:01
Comment thread webserver/logger/logger.py
Comment thread webserver/app.py
Copy link
Copy Markdown
Contributor

@marconetsf marconetsf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implementação congruente, módulo logger bastante util. Apesar de ter vários arquivos modificados a maioria das modificações relevantes acontecem nos arquivos bufferhandler, formatter, logger, parser, bem como no lado do runtime no log.c

Base automatically changed from RTOP-52-Standard-Network-Driver to development October 8, 2025 12:34
@lucasbutzke lucasbutzke merged commit 19252e0 into development Oct 8, 2025
@lucasbutzke lucasbutzke deleted the RTOP-76-Runtime-logs-parser branch October 8, 2025 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants